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ABSTRACT 



The final versions of nationally developed standards for 
science education have recently been published for use by practitioners, and 
it is obvious that these standards will have some influence on how science 
education is conducted. The effects of the standards on teaching and 
curriculum development can be anticipated, but their effects on evaluation in 
science education are not as obvious, although it does follow that 
instruments developed for evaluating a specific program that are true to the 
standards should be applicable to evaluation of other science education 
programs. Some evaluation tasks related to relevant standards found in recent 
literature are listed and described for STEP, a science education staff 
development program. Data collection instruments usually flowed readily from 
the standards, although their construction often required careful reading and 
interpretation. The STEP project was able to produce standards -based 
instruments such as Likert- type self-report rating scales, open-ended 
interview schedules, observation instruments, and program evaluation 
checklists. Some examples are given for: (1) teacher feedback on classroom 

teaching; (2) teacher feedback on effectiveness of a teacher institute; and 
(3) observational data collection during a training institute. The 
standards -based approach to evaluating a standards -based program seems to be 
worthwhile and efficient. An evaluation form is attached. (Contains 14 
references . ) (SLD) 



******************************************************************************** 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 




(Science) Education Standards Yield Standard Data-Collection Instruments 



ON 

<N 

O 



s 



PERMISSION TO REPRODUCE AND 
disseminate this material 

HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
information center (ERIC) 



y.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCi^lONAL RESOURCES INFORMATfON 
CENTER (ERIC) 

B^his document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Morris K. Lai & Donald B. Young 
Curriculum Research and Development Group 
University of Hawai‘i 



> 

N 



rt 

o 







Paper presented at the annual meeting of the 
American Educational Research Association 
Session 23.04 
26 March 1997 



Chicago, Illinois 



2 



(Science) Education Standards Yield Standard Data-Collection Instruments 
Overall approach 

Only recently have the final versions of nationally developed standards for science education been 
published (e.g., the National Science Education Standards (NSES) released by the National 
Research Council in late 1995) for use by practitioners. Some associations are still developing their 
version of science education standards (e.g., the Association for the Education of Teachers of 
Science), some have posited principles or models (e.g.. The National Institute for Science 
Education), some have developed frameworks based on the NSES (e.g.. National Science 
Teachers Association’s A Framework for High School Science Education (1996)), and some have 
published benchmarks (e.g., American Association for the Advancement of Science (AAAS), 
1993). Also relatively recent are various professional development standards (e.g.. National Staff 
Development Council (1994) and The Program Evaluation Standards (1994) sponsored by AERA 
and 14 other major educational organizations. 

It is obvious that these standards will have some influence on how science education is conducted. 
In a June/July 1996 article in the Educational Researcher, Linn and Muilenburg noted “Recently 
published standards for science, mathematics, history, and other fields challenge educators to 
design instruction that set in motion a lifelong habit of learning.” This requires, they go on to say, 
the redesign of instruction. 

Although many believe the standards will help improve the way they teach, they expect to face a 
variety of obstacles (the top three being (a) time for planning, (b) financial support for professional 
development, and (c) materials, resources, and facilities), reported Meg Sommerfield in the April 
10, 1996 issue of Education Week. Her findings were based on a survey sent to a random sample 
of members of the National Science Teachers Association. 

Effect of standards on evaluations of science education programs 

Not as obvious as the effects of the standards on teaching and curriculum development are the 
standards’ effects on evaluation in the field. It is noteworthy to realize that if one buys into the 
science education standards, then it directly follows that instruments developed for evaluating a 
specific program that are true to the standards should be essentially applicable to evaluation of other 
science education programs. Just as is the case for curriculum development, however, certainly 
different versions of standards-based data-collection instruments can emerge. 




Here we show a listing of the major relevant standards found in the recent literature. Evaluation 
tasks (ET_) for STEP, a science education staff development program, are given in the second 
column and are subsequently described in more detail. 



Source of Standards 



American Association for the Advancement of 
Science (AAAS), Benchmarks for Science 
Literacy (1 1/93) [Also Science for All 
Americans, 1989] 

Education Department General Administrative 
Regulations (EDGAR), 1994 revision 

National Center for Improving Science 
Education (NCISE) [In Promising Practices] 



National Research Council, *National Science 
Education Standards, 1996. Teaching, 
Professional Development, and Assessment. 

National Staff Development Council (NSDC), 
Standards for Staff Development, 1994 

Program Effectiveness Panel (U.S. Dept, of 
Education) guidelines (e.g., as outlined in 
Making the Case, 1988) 

Program Evaluation Standards (AERA, NEA, 
& many others), 1994 

Other Standards (less emphasis on) 



Evaluation Task(s) or Rationale 



ET4, ETIO, ET13; Most states use these 
standards. 



Overall evaluation and future proposals 



ESTEEM (1995) instruments to measure 
teaching practices and students outcomes. 

“...no other constructivist classroom 
observation instrument.” ET6, ET7, ET8 (?) 

ETl, ET3, ET4, ET5, ET6, ET7, ET8 (?), 
ETIO, ET12. Also for system, program, and 
content. Our focus is on the teaching standards. 

ET5, ET6, ETIO 



ET2, ET4, ET11,E12, E13 



Overall STEP evaluation design. ET9 



E.g., US Dept, of Education (1994 draft), 
ASCD (11/94 draft), AETS. 



*Available at http://www.nap.edu/nap/online/nses/FrontDoors.html 

The ETs refer to the following delineated list of 13 evaluation tasks that constitute the essential 
tasks of the evaluation. 
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STEP Evaluation Tasks 



ETl. Videotape teacher’s rendition of best lesson before training and one year later — grades 4 
and/or 5 

• Identify 10 sites; 2 teachers per site; total 20 teachers 

• Identify local coordinator to videotape to standardize data collection 

• Develop protocol for videotaping and send out 

• Develop selection criteria (If more than 2 teachers at site, random selection; 
length of lesson, tape format, etc.) 

• Contract for IOTA analysis of tapes 

• Conduct live IOTA analysis and tape analysis to establish reliability 

• Develop NRC National Science Education Teaching Standards Checklist; contract an 
external evaluator to conduct the analyses 

• Identify teacher at same sites for post taping only 

• Tape and analyze best lesson one year later 



ET2. Case studies, grade 4 

• Clarify status of single case studies 

• Complete Hawai‘i case studies (2) following established guidelines 

• Test Speitel’s program for qualitative analysis and videotape analysis; use 
as appropriate 

• Contract an external evaluator to conduct cross case analysis 

• Conduct cross case analysis 

• Link to content and teaching standards; which are addressed? 



ET3. Videotape best lesson before training and one year later — grades 7-9 

• Identify 5 rural sites and 5 urban sites 

• Identify local coordinator type to videotape to standardize data collection 

• Send out protocol for taping 

• Develop selection criteria 

• Identify teacher at same sites for post taping only 

• Tape and analyze best lesson one year later 

ET4. Test data collection on FAST 

• Analyze National Assessment of Educational Progress (NAEP) items 

• Administer Landscape Survey (NAEP items) grades 8 and 9 in Cincinnati 

• Identify possible comparison groups within schools 

• Collect school impact data with comparison groups on grades, attendance, cuts, electing 
more science, etc. 



ET5. Impact on teaching after training and first year implementation 

• Revise instrument based on pilot 

• Revise methodology based on pilot 

• Establish criteria for observation (grade, length of observation, number of sites) 
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ET6. Institute Data— DASH, FAST, HMSS 

• Review and summarize participant evaluation data from summer ‘95, ‘96, and ‘97. 

• by institute 

• by state 

• Review & revise free response to standards-based instruction 

• Use instruments at end of institutes 

• Apply revised RFI instrument with revised methodology 

ET7. Self-rating instruments — DASH, FAST, HMSS 

• Develop NRC teaching standards checklist 

• Develop standards-based free response instrument 

• Administer NRC teaching standards instrument first day of instruction at all sites 

• Randomly select 150 teachers for post follow up 



ET8. Artifacts as evidence of achievement 

• Adapt or develop artifact review procedures 

• Develop criteria for selecting teachers/classes for artifacts 

• DASH — Collect artifact data (actual; photo; videotape) grade 4 and 5. Evaluate 
using criteria. Have some grade 4. 

• FAST — Collect student notebooks as portfolios. Evaluate using criteria 



ET9. Collect data addressing evaluation standards 

ETIO. Collect teacher institute data addressing 

• Objectives 1-3; Numbers of new trainers trained, numbers of teachers trained, 
numbers of teachers provided support services 

• Objective 4. Level of achieving standards 

• Correct and revise RFI draft progress report 

• Repeat data collection and reporting 



ETl 1. Existing achievement test data at implementation sites 

• Adapt or develop new form for processing achievement data 
(Standardized test data by class/school; school demographics; comparison data; 
performance testing; reading/mathematics scores; other indicators of impact) 

• Solicit partners to systematically collect data 

• Collect existing data 

• Categorize, analyze, report 

ETl 2. Teachers as leaders data 

• Collect and catalog all indicators or developing leadership. (Awards, anecdotes; 
professional meetings attended; presentations at professional meetings; within school faculty 
development; becoming certified trainers; action research reports; supervisor testimonials; 
enrolling for advanced degree; school, district, state, or professional committees; publications; 
newspaper reports; other) 

• Survey institute participants 

• Compile data and report 
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ET13. HMSS 

Design and collect impact data as appropriate (this curriculum has not been as extensively 
evaluated as have DASH and FAST) 

Standards-based data-collection instruments 

Data-collection instruments usually readily flowed directly from the standards but often required a 
careful reading and interpretation of the standards. With some modest modifications of wording 
and scope, we were able to produce standards-based instruments such as Likert-type self-report 
rating scales, open-ended interview schedules, classroom and teacher training observation 
instruments, and overall program evaluation checklists. Here we give some specific examples. 

1. Teacher feedback on classroom teaching 

In the original statement of the standards, examples of what teachers do to carry out each standard 
are given. We used these examples to create a Likert-type scale for teachers to give feedback on , 
their classroom teaching (as opposed to their feedback on the effect of the teacher-training 
workshop). We selected from the examples those which we used virtually verbatim if that made 
sense or did some modest editing as we felt was appropriate. Here we give an example of each 
type: 

NRC original wording 

“Work together as colleagues within and across 
disciplines and grade levels.” 



“Encourage and model the skills of scientific 
inquiry, as well as the curiosity, openness to 
new ideas and data, and skepticism that 
characterize science.” 



Wording for Likert scale (strongly 
agree to strongly disagree) 

“I work together with my colleagues within and 
across disciplines and grade levels.” (may want 
to write two or three separate items so as to 
avoid multiple stem — see example that follows) 

“In my class, I encourage the skills of scientific 
inquiry.” 

“In my class, I model the skills of scientific 
inquiry.” 

“In my class, I encourage skepticism that 
characterizes science.” 

“In my class, I model skepticism that 
characterizes science.” 
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2. Teacher feedback on effectiveness of teacher institute 



First we give an example of a data-collection instrument that can be used to get teacher feedback on 
how effective the institute was in addressing the standards: 

Usefulness of the STEP Institute in Meeting Science Education Standards 

(Room for teachers’ responses have been reduced for this paper.) 

Please give an example of how you could use what you learned in this STEP Institute to address 
the following science education standards (use the back of this sheet if more space is needed); 

1 . Teachers of science plan an inquiry-based science program for their students. 

2. Teachers of science guide and facilitate learning. 

3. Teachers of science engage in ongoing authentic assessment of their teaching and of student 
learning. 

4. Teachers of science design and manage learning environments that provide students with the 
time, space, and resources needed for learning science. 

5. Teachers of science develop communities of science learners that reflect the intellectual rigor of 
scientific inquiry and the attitudes and social values conducive to science learning. 

6. Teachers of science actively participate in the ongoing planning and development of the school 
science program. 

7. Science education is accessible to all students 

8. Science learning builds on students’ prior experience and knowledge. 

9. Teachers help students relate to personal and social needs. 

Items 1-6 are based on the teaching standards from the National Research Council (NRC). In most 
cases, the exact wording of the NRC standard itself was used (e.g., items 1, 2, 4, 5, and 6). For 
item 3, we added the word “authentic” to reflect a standard from the National Center for Improving 
Science Education’s (NCISE) list (“Use authentic assessments to chart teaching and learning.”). 
Thus item 3 represents both the NRC and the NCISE standard. 

3. Observational data collection during training institute 

We developed an observation instrument based on staff development standards. As expected, it 
was somewhat more difficult to develop an instrument in this area. The instrument we developed 
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for the staff development institutes has observers rate standards (e.g., as follows: “Observed — 
Clear focus; Observed — Adequately addressed; Observed — Somewhat addressed; Not observed”). 

Inappropriateness of some standards for developing items for immediate feedback 

In our attempt to directly address some of the standards, we found that a number were longitudinal 
in nature or were to be addressed at the teachers’ schools rather than at a workshop. That made us 
rethink our evaluation design in the following way. We are asking teachers to videotape their “best 
lesson” before the workshop. We will ask them to do the same a year later. These videotapes will 
then be analyzed using IOTA and science education teaching standards from the National Research 
Council. We also had them fill out before the start of the workshop a Likert-type scale addressing 
how they taught in their classroom. We will give them the same questionnaire during the end of the 
current school year. 

Use of method to design and conduct other evaluations 

We found the standards-based approach to evaluating a standards-based program to be a 
worthwhile one. To the extent that professional standards are accepted in a given field, it should be 
possible to develop evaluation data-collection instruments that, with modest adjustments, can be 
valid for most evaluation efforts. Not only would this approach be relatively efficient, but it would 
also have the benefit of being based on outcomes and processes that the leading professional 
organizations in the field have deemed as standards worthy of addressing and meeting. Two 
sample data-collection instruments are attached. 
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Usefulness of the STEP Institute in Meeting NRC Science Education Standards 

Please give an example of how you could use what you learned in the STEP Institute to address 
the following science education teaching standards (use the back of this sheet if more space is 
needed): 

1. Teachers of science plan an inquiry-based science program for their students. 



2. Teachers of science guide and facilitate learning. 



3. Teachers of science engage in ongoing authentic assessment of their teaching and of student 
learning. 



4. Teachers of science design and manage learning environments that provide students with the 
time, space, and resources needed for learning science. 



5. Teachers of science develop communities of science learners that reflect the intellectual rigor of 
scientific inquiry and the attitudes and social values conducive to science learning. 



6. Teachers of science actively participate in the ongoing planning and development of the school 
science program. 



Q © Curriculum Research & Development Group, University of Hawai'i 



ERIC 







Self-Report about Teaching 

Name School Date 

Directions; For the following statements, circle “SA” is you strongly agree, “A” if you agree, 
“D” if you disagree, or “SD” if you strongly disagree. 



1. 


I have developed a framework for yearlong and short-term 
science education goals for my students. 


SA 


A 


D 


SD 


2. 


I select teaching and assessment strategies that nurture a 
community of science learners. 


SA 


A 


D 


SD 


3. 


I work together with my colleagues within and across disciplines 
and grade levels. 


SA 


A 


D 


SD 


4. 


I use an inquiry-based approach to teaching science. 


SA 


A 


D 


SD 


5. 


I orchestrate discourse among my students about scientific ideas. 


SA 


A 


D 


SD 


6. 


I challenge my students to accept and share responsibility for 
their own learning in science. 


SA 


A 


D 


SD 


7. 


In my class, I encourage the skills of scientific inquiry. 


SA 


A 


D 


SD 


8. 


In my class, I model the skills of scientific inquiry. 


SA 


A 


D 


SD 


9. 


In my class, I encourage skepticism that characterizes science. 


SA 


A 


D 


SD 


10. 


In my class, I model skepticism that characterizes science. 


SA 


A 


D 


SD 


11. 


I analyze assessment data to guide my science teaching. 


SA 


A 


D 


SD 


12. 


I guide my students in self-assessment. 


SA 


A 


D 


SD 


13. 


I use student data, observations of teaching, and interactions 
with colleagues to reflect on and improve my teaching practice. 


SA 


A 


D 


SD 


14. 


I structure the time available so that my students are able to 
engage in extended investigations. 


SA 


A 


D 


SD 


15. 


For my science teaching, I identify and use resources outside the 
school. 


SA 


A 


D 


SD 


16. 


I engage my students in designing the science learning 
environment. 


SA 


A 


D 


SD 


17. 


I nurture collaboration among my students. 


SA 


A 


D 


SD 


18. 


I have helped plan and develop my school’s science program. 


SA 


A 


D 


SD 


19. 


I have participated fully in planning and implementing my 
professional growth and development strategies. 


SA 


A 


D 


SD 




© Curriculum Research & Development Group, University of Hawai‘i 
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